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Abstract 


This  paper  presents  a  view  of  the  state  of  the  art  in  cybersecurity  monitoring  technology. 
The  paper  develops  the  view  from  six  sources:  three  prior  reports  (two  national,  one 
MITRE),  a  survey  of  commercially  available  software,  a  survey  of  government  software,  and 
a  survey  of  government-funded  research  projects.  The  author  performed  the  surveys  for  this 
paper. 
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Preface 


A  couple  of  years  ago,  I  started  collecting  information  about  intrusion  detection  tools  and 
projects.  After  a  while,  I  noticed  that  I  was  including  tools  that  were  not  inherently  intrusion 
detection  tools  because  they  were  closely  related  to  intrusion  management  in  one  form  or 
another.  For  example,  vulnerability  scanners  can  assist  in  making  it  more  difficult  for  an 
attacker  to  succeed.  This  situation  created  a  naming  problem:  by  what  category  name  should 
I  refer  to  these  closely  related  tools? 

Unfortunately,  as  noted  by  several  authors  over  the  past  few  years,  there  is  no  common 
vocabulary  for  talking  about  the  technical  area  that  encompasses  intrusion  detection, 
vulnerability  scanning,  security  policy  compliance  monitoring,  and  related  topics.  I  decided 
to  use  terminology  that  made  sense  to  me,  based  on  the  usual  meanings  of  words  as 
described  in  modern  dictionaries.  It  was  this  approach  that  gave  rise  to  my  use  of  the  word 
“anomaly”  to  refer  to  anything  out  of  the  ordinary,  normal,  or  expected  in  the  configuration 
and  operation  of  a  network  and  the  components  within  or  attached  to  it. 

However,  language  usage  often  ignores  logic.  Infosec  professionals  generally  use  the 
word  “anomaly”  in  a  restricted  technical  sense  to  mean  what  I  would  call  statistical  deviation 
detection.  Like  it  or  not,  “anomaly”  refers  to  deviant  user  behavior,  not  deviant  anything  else. 
In  addition,  I  came  to  realize  that  not  all  tools  of  interest  deal  with  anomalies  anyway.  I 
decided  several  months  ago  it  was  time  to  find  a  better  word  or  phrase! 

Accordingly,  I  asked  subscribers  to  the  Infosec  e-mail  list  for  suggestions.  I  got  14 
responses,  mulled  them  for  a  while,  then  made  a  selection.  The  terminology  “CyberSecurity 
Management  and  Monitoring  Tools”  seemed  best  to  cover  most  of  the  ideas  that  were 
offered.  This  phrase  is  based  on  the  core  idea  of  "management  and  monitoring  tools"  for 
information  safety  in  computers  and  computer  networks.  To  distinguish  such  tools  from 
network  management  and  monitoring  tools,  "Security"  is  added.  To  distinguish  the  kind  of 
security  these  tools  deal  with  from  physical  security,  "Cyber"  is  added.  Influenced  by 
modern  object-naming  terminology,  CyberSecurity  is  spelled  with  two  capital  letters. 

I  quickly  discovered,  however,  that  I  was  uneasy  with  the  new  terminology.  Monitoring, 
like  many  other  relevant  activities,  is  just  one  of  many  functions  that  fit  the  category 
CyberSecurity  Management.  The  category  “CyberSecurity  Management”  covers  a  wide  array 
of  capabilities,  including  CyberSecurity  Monitoring.  The  train  of  thought  I  have  just 
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described  led  to  the  terminology  used  in  this  revision  as  well  as  in  the  revision  of  the 
compendium1  of  tools  and  projects. 

The  reader  should  note  that  this  revision  only  updates  terminology.  The  estimation  of  the 
state  of  the  art  as  I  understood  it  in  July  1999  has  not  been  revised.  The  reader  should  refer  to 
the  revised  supplement,  listed  in  the  bibliography  of  this  paper,  for  an  update  on  the  state  of 
the  art,  as  of  February  2000. 

August  23,  2000 


1  The  revision  is  listed  in  the  bibliography  of  this  paper.  The  reference  list  still  refers  to  the 
old  compendium  because  only  the  information  in  it  was  used  in  arriving  at  the  state  of  the 
art  description  in  this  paper. 
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State  of  the  Art  in  CyberSecurity  Monitoring 


Introduction 

This  paper  synopsizes  the  state  of  the  art  in  cybersecurity  monitoring  systems.  Having 
said  that  necessitates  some  explanation  about  what  we  mean  by  “cybersecurity  monitoring” 
and  what  we  mean  by  “state  of  the  art.” 

About  CyberSecurity  Monitoring  Systems 

We  are  interested  in  automated  capabilities  that  can  detect  intrusions  or  other 
abnormalities  in  computer  systems,  report  them  in  useful  ways,  remove  discovered 
anomalies,  and  repair  damage  they  may  have  caused.  Included  in  this  scope  of  interest  are 
traditional  intrusion  detection  and  reaction  tools.  The  broader  scope  of  CyberSecurity 
Monitoring  also  includes  vulnerability  scanners,  infraction  scanners,  and  security  compliance 
monitors.  These  tools  protect  not  only  against  intruders  but  against  errors  and  carelessness  in 
administration  and  operation  of  end  systems  and  network  components. 

Even  within  the  narrower  scope  of  intrusion  prevention,  one  can  fashion  protection 
against  cyber  intruders  within  a  spectrum  of  techniques.  At  one  end  of  the  spectrum  is  the 
method  of  detecting  intruders.  In  this  method,  one  uses  intrusion  detection  tools  to  watch 
what  is  going  on  in  the  network  to  discover  suspicious  events.  If  perfect  intrusion  detection 
and  reaction  systems  were  available,  there  might  be  no  need  for  any  other  measures  to 
protect  against  cyberattack.  At  the  other  end  of  the  spectrum  is  the  method  of  ensuring  that 
all  the  components  of  the  network,  including  firewalls,  routers,  servers,  and  workstations,  are 
equipped  to  fully  repel  any  attack.  In  this  method,  one  would  not  try  to  detect  intrusive 
connections  coming  from  outside  one’s  network  since  they  can  do  no  harm.  In  theory,  even  a 
denial  of  service  attack  could  be  thwarted  in  this  way  because  the  components  of  the  data 
communications  infrastructure  would  be  smart  enough  not  to  carry  the  traffic  that  would 
cause  the  denial  of  service.  Of  course,  this  is  just  wishful  thinking  since  we  do  not  know  how 
to  make  the  components  so  smart  as  to  be  able  to  do  this.  In  practice,  what  can  be  done 
throughout  the  theoretical  spectrum  should  be  done  to  provide  the  best  protection  for 
investment  made.  Prudence  demands  balance:  organizations  should  properly  set  up  and 
configure  the  components  of  their  networks  using  current  best  practices  and  they  should 
provide  state  of  the  art  intrusion  detection  capability,  depending  wholly  on  neither  to 
adequately  protect  their  computing  resources.  The  capabilities  brought  to  bear  by  the 
nondetection  tools  protect  not  only  against  intruders  but  against  errors  and  carelessness  in 
administration  and  operation  of  both  end  systems  and  network  components. 

Doing  these  things  is  not  a  one-time  chore.  Network  topologies  tend  to  be  dynamic. 

Often  it  is  difficult  to  control  the  comings  and  goings  of  hosts  on  a  network,  especially  in 
large  networks.  The  job  of  properly  setting  up  and  configuring  components  often  requires 
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skilled  personnel,  who  are  in  short  supply.  In  addition,  new  cyberattacks  may  demand  new 
protections  or  responses. 

Prudent,  affordable,  continuous  protection  of  one’s  network  involves  monitoring  the 
network  for  anomalies  of  various  kinds,  whether  they  are  suspicious  textual  strings  in  a 
network  packet  or  undesirable  values  for  important  keys  in  NT  registries.  Moreover,  it 
involves  correcting  detected  anomalies,  whether  that  means  terminating  a  connection  or 
reconfiguring  a  server. 

We  call  an  automated  system  that  performs  or  assists  in  such  tasks  a  CyberSecurity 
Monitoring  (CSMn)2  system.  Besides  checking  network  packets  for  suspicious  strings,  or 
monitoring  a  user’s  behavior  looking  for  deviations  from  an  established  pattern,  the  CSMn 
system  checks  components  of  the  network  for  errors  of  omission,  misconfigured 
applications,  and  errors  in  system  parameters.  When  the  CSMn  system  finds  an  abnormality, 
it  reacts,  generally  by  trying  to  fix  the  abnormality.  Its  response  may  be  restricted  to  issuing 
an  alert  for  certain  conditions.  For  others,  it  may  be  able  to  fully  correct  the  problem.  In  some 
cases,  it  may  be  able  to  provide  ancillary  information  that  will  assist  an  administrator  in 
correcting  the  problem.  What  it  can  do  will  be  determined  by  the  state  of  the  art,  the  budget, 
and  the  information  operation  to  be  protected. 

Besides  budgetary  considerations,  the  extent  of  the  protection  domain  determines  the 
needed  capacity  of  the  CSMn  system  for  that  domain.  Networks  tend  to  grow,  thereby 
extending  the  scope  of  interest  for  a  CSMn  system.  Thus,  scalable  CSMn  systems  are 
needed,  not  only  so  that  the  same  basic  system  can  serve  domains  of  different  size,  but  also 
so  that  it  can  accommodate  significant  growth  in  the  domain  it  protects. 

The  appendix  in  this  report  contains  a  discussion  of  related  ideas  on  cybersecurity 
monitoring,  describing  methods  of  intrusion  and  anomaly  detection  and  providing  a 
classification  of  CSMn  tools. 

About  Describing  the  State  of  the  Art 

A  description  of  the  state  of  the  art  in  some  technology  generally  includes  more  than 
what  has  been  reduced  to  ordinary  practice.  State-of-the-art  reports  may  use  research  efforts, 
for  example,  to  describe  the  level  of  sophistication  achieved  in  some  developing  technology, 
such  as  the  technology  that  deals  with  nanocomputers.  A  danger  is  that  one  may  take  as 
“reduced  to  practice”  what  is  merely  a  possibility.  We  too  will  include  more  than  “practice” 
in  this  report,  but  we  will  focus  on  what  is  commercially  available  today  in  cybersecurity 
monitoring.  Besides  wanting  to  avoid  confusing  “practice”  with  “theory”,  we  are  motivated 
by  a  concern  for  the  particular  needs  of  our  military  customers.  We  want  a  pragmatic  view 


2  We  use  the  acronym  “CSMn”  instead  of  “CSM”,  reserving  the  latter  to  mean 
“CyberSecurity  Management”. 
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that  is  relevant  to  the  Air  Force’s  mission-oriented  networks  and  computers.  This  view  will 
enable  us  to  tell  what  is  currently  available  that  can  provide  an  information-assurance  benefit 
while  fitting  within  the  constraints  of  that  environment. 

We  have  drawn  on  three  reports  dealing  with  state  of  the  art,  which  we  will  identify 
shortly.  These  reports  deal  with  intrusion  detection  and  reaction,  not  the  broader 
CyberSecurity  Monitoring  of  interest  here.  Thus,  they  do  not  consider  vulnerability  scanners, 
infraction  scanners,  and  security  compliance  monitors. 

We  present  a  broader  picture  in  this  synopsis  and  bring  it  up  to  date  by  summarizing  what 
we  have  learned  from  current3  descriptions  of  commercial  off-the-shelf  (COTS)  products. 
This  part  of  our  synopsis  provides  the  pragmatic  view  we  talked  about  earlier.  Government 
off-the-shelf  (GOTS)  products  also  help  define  the  current  state  of  the  art  because  they  may 
provide  capabilities  not  yet  found  in  commercial  products  or,  at  least,  their  use  may  shed 
light  on  the  economics  affecting  their  users.  In  addition  to  a  summary  of  GOTS  products,  we 
include  a  look  ahead  by  summarizing  what  we  know  of  current  research  efforts  funded  by  the 
U.S.  government.  Note  that  this  paper  does  not  include  information  about  freeware, 
shareware,  research  by  vendors,  ad  hoc  consortia  or  teams  that  may  be  implementing  CSMn 
tools,  or  any  other  source  not  explicitly  identified  herein.  Having  so  noted,  readers  should 
understand  the  exact  scope  of  the  state  of  the  art  as  reported  here  and  be  able  to  judge  the 
applicability  of  this  synopsis  for  each  situation  they  may  deal  with. 

In  short,  this  synopsis  draws  on  these  sources  of  information 

•  The  National  Info-Sec  Technical  Baseline  report  on  intrusion  detection  and  response 

[1] 

•  The  description  of  the  state  of  the  art  in  network-based  intrusion  detection  systems  in 
a  report  of  Hill  and  Aguirre  [2] 

•  The  report  of  the  Intrusion  Detection  Subgroup  of  the  National  Security 
Telecommunications  Advisory  Committee  on  the  implications  of  intrusion  detection 
technology  research  and  development  on  national  security  and  emergency 
preparedness  [3] 

•  Product  descriptions  of  commercial  off-the-shelf  (COTS)  and  government  off-the- 
shelf  (GOTS)  CSMn  systems  [4] 

•  Descriptions  of  current  research  in  cybersecurity  monitoring  [4] 

How  This  Paper  is  Organized 

We  begin  by  summarizing  the  conclusions  from  the  three  reports,  in  chronological  order. 
Then  we  take  a  look  at  commercial  products,  government  products,  and  research  efforts. 

3  As  of  March,  1999 
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Finally  we  capture  the  state  of  the  art  as  a  summary  of  the  preceding  information  sources. 
Thus,  the  rest  of  this  paper  is  organized  as  follows: 

•  National  Info-Sec  Technical  Baseline:  summary  of  findings 

•  Report  of  Hill  and  Aguirre:  summary  of  findings 

•  Intrusion  Detection  Subgroup’s  Report:  summary  of  findings 

•  Commercial  Products:  summary  of  product  types  and  characteristics 

•  Government  Products:  summary  of  product  types  and  characteristics 

•  Research  Efforts:  summary  of  principal  lines  of  investigation 

•  Summary:  a  capsule  description  of  the  state  of  the  art  in  CyberSecurity  Monitoring 

National  Info-Sec  Technical  Baseline 

In  the  Executive  Summary  of  the  baseline  report,  the  1996  state  of  the  art  in  intrusion 
detection  is  succinctly  characterized  in  the  first  paragraph 

“The  state  of  the  art  in  logical  intrusion  detection  of  national  information 
infrastructure  (Nil)  systems  is  such  that  a  human  expert  working  with  a  well- 
developed  set  of  tools  can  implement  a  detection  system  in  a  few  months  for  a 
special-purpose  computing  environment  with  the  properties  that  it 

(1)  reliably  detects  a  substantial  number  of  known  intrusion  techniques, 

(2)  detects  substantial  short-term  changes  in  user  and  system  behavior, 

(3)  produces  many  alarms  that,  on  investigation,  are  not  intrusions  (false-positives),  and 

(4)  fails  to  alarm  on  an  unknown  number  of  intrusions  (false-negatives).” 

Similarly,  the  report  summarizes  the  situation  in  automated  response  as  follows: 

“The  state  of  the  art  in  automated  response  to  detected  intrusions  is  that  we  can 
program  a  wide  variety  of  responses,  ranging  from  increased  defenses  to 
offensive  counter-strikes.  Examples  throughout  this  range  have  been 
demonstrated.  Several  important  automated  response  issues  remain  essentially 
unaddressed  at  this  time,  including,  but  not  limited  to,  (1)  limiting  the  effect  of 
automated  response  so  as  to  prevent  cascade  and  livelock  failures  that  may  be 
caused  by  the  response  system,  (2)  providing  safeguards  against  false-positives 
and  enemy-induced  erroneous  responses,  and  (3)  using  the  response  system  to 
push  the  point  of  attack  deflection  back  toward  the  attack  source.” 

The  report  identifies  these  practical  issues 

•  Testing  is  lacking,  with  most  systems  rarely  tested  beyond  demonstrating  that  they 
detect  some  anomalous  events 
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•  Damage  assessment  and  recovery  mechanisms  are  lacking  in  the  vast  majority  of 
systems 

•  Scalability  of  IDR  systems  is  severely  limited.  To  be  of  significant  utility  in  modern 
information  environments,  IDR  systems  must  be  capable  of  handling  large  numbers 
of  events.  By  contrast,  most  of  the  systems  examined  by  the  reporters  detect  events  at 
the  system  level.  Work  directed  toward  correlating  activities  between  systems  is  in 
the  very  early  stages  and  much  more  work  needs  to  be  done. 

The  report  stresses  the  importance  of  identifying  the  source  of  an  attack  and  states  that 
traceback  capability  is  most  difficult  in  the  computer-networking  environment. 

“In  today’s  Internet,  there  are  no  central  controls  or  methods  to  trace  through  the 
infrastructure  without  the  cooperation  of  a  potentially  large  number  of 
independent  infrastructure  providers.  In  recent  years,  some  limited  tools  have 
been  developed  to  try  to  trace  an  intrusion  to  a  source,  but  these  tools  are  only 
effective  against  the  least  sophisticated  attackers.  They  do  not  allow  traceback 
when  IP  address  forgery  is  used,  when  intermediate  nodes  fail  to  cooperate4, 
when  firewalls  block  further  traceback,  or  when  the  intruder  breaks  into  an 
intermediate  site  in  order  to  launch  the  attack.”5 

Report  of  Hill  and  Aguirre 

The  state  of  the  art  for  network-based  intrusion  detection  was  described  in  September 
1997  by  Hill  and  Aguirre.  [7]  In  summary,  they  reported  that6,  for  network-based  intrusion 
detection, 

•  No  common  terminology  or  taxonomy  exists  for  discussing  intrusion  detection 
techniques  or  implementation 

•  Current  network-based  intrusion  detection  systems  (IDSs)  are  signature  recognition 
systems 

•  The  IDSs  can  be  modeled  as  being  performed  by  three  agents 
-  Collection  function 


4  We  note  that  traceback  techniques  that  do  not  necessarily  require  the  cooperation  of  the 
independent  infrastructure  providers  have  a  much  better  chance,  technically  speaking,  to 
locate  the  cybersource  of  an  attack;  the  United  States  Justice  Department,  however, 
typically  does  not  allow  military  departments  to  use  such  techniques. 

5  The  report  cites  Cohen’s  paper  A  Note  on  Distributed  Coordinated  Attacks ,  Computers 
and  Security,  August  1996. 

6  As  of  September  1997. 
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-  Analysis  function 

-  Management  function 

•  The  collection  function  needs  improvement:  it  should  address  new  physical  media, 
such  as  ATM  and  Fast  Ethernet,  provide  more  flexibility  in  signature  specification, 
and  understand  more  application  protocols. 

•  Better  analysis  support  is  even  more  urgent  a  need: 

-  Improved  expert  systems  should  reduce  the  false  alarm  rate 

-  Better  distribution  of  problem  identification  is  needed 

-  Tools  that  can  correlate  activity  from  distributed  sources  and  activity  directed 
toward  distributed  targets  are  crucial  to  address  issues  of  information  warfare. 

•  There  is  growing  recognition  that  there  would  be  high  utility  in  integrating  the  output 
of  different  entities  involved  in  network  security,  including  routers,  firewalls, 
proxies,  and  host-based  and  network-based  IDSs. 

Intrusion  Detection  Subgroup’s  Report 

This  study  focused  on  influencing  research  and  development  of  intrusion  detection 
technology  so  that  national  security  and  emergency  preparedness  (NS/EP) 
requirements  can  be  satisfied.  The  findings  of  the  subgroup  on  technology  relevant  to  the 
telecommunications  infrastructure  were 

•  Research  and  development  seldom  focuses  on  controlling  elements  of  the 
telecommunications  infrastructure;  IDSs  in  use  mostly  were  developed  to  meet  the 
needs  of  host  systems  in  unique  environments  (for  example,  UNIX,  Novell,  Microsoft 
NT) 

•  “Many  of  the  deployed  IDSs  are  unable  to  scale  to  the  large  environments 
characterized  by  thousands  of  network  nodes,  which  limits  their  ability  to  detect 
intrusions  effectively  across  different  platforms,  applications,  networks,  and 
infrastructures.” 

•  Standards,  testing,  and  validation  procedures  are  needed  to  enable  verification  of 
IDSs’  capabilities;  the  development  and  use  of  standard  metrics  would 
be  helpful  in  this  area 

•  False  alarms  are  a  major  performance  problem  in  current  IDSs 

•  The  fact  that  current  IDSs  can  detect  only  what  they  have  been  preprogrammed  to 
detect,  like  virus  detectors,  limits  their  effectiveness  in  the  face  of  new 
attacks 

•  Detection  and  response  need  to  occur  in  real  time  to  limit  potential  damage;  current 
IDSs  fall  far  short  of  this  ideal 

•  No  systems  appeared  to  provide  an  automated  damage  assessment  and  response 
capability 
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Commercial  Products 

We  base  our  findings  in  this  section  on  product  information  gathered  primarily  from 
vendors’  web  sites,  as  reported  in  the  Compendium  of  Tools  and  Projects  [4].  The 
Compendium  includes  information  about  397  commercially  available  products  from  a 
number  of  vendors.  In  the  Compendium,  vendors  have  been  grouped  as  primary  providers, 
secondary  providers,  and  others.  Primary  providers  are  those  with  the  highest  revenues  as 
reported  in  the  Hurwitz  Group  white  paper  Information  Security:  Assessing  Risks  and 
Detecting  Intrusions.  Secondary  providers  are  those  with  comparable,  competitive  tools  or 
systems,  as  identified  in  the  same  paper.  Other  vendors  offering  CSMn  tools  have  also  been 
included. 

The  primary  providers  are 

•  AXENT  Technologies 

•  CISCO  (recently  acquired  WheelGroup) 

•  Internet  Security  Systems  (ISS) 

•  Intrusion  Detection,  Inc.,  a  Security  Dynamics  Company 

•  Network  Associates,  Inc.  (recently  acquired  Trusted  Information  Systems  (TIS)) 

•  PLATINUM  Technology 

As  is  suggested  from  the  list  of  names,  the  primary  providers  tend  to  be  large,  well- 
established  vendors  who  have  either  been  in  the  intrusion  detection  area  for  a  while  or  have 
acquired  significant  capability  through  acquisition.  These  same  vendors  tend  to  have  the 
most  recent  products  or,  in  some  cases,  recently  upgraded  versions  of  older  products. 

Summarized  Information  About  the  Tools  in  the  Compendium 

The  next  three  tables  summarize  selected  information  from  the  compendium.  Table  1 
shows  counts  of  the  number  of  products  organized  by  type  of  product  and  category  of  vendor 
(P:  Primary;  S:  Secondary;  O:  Other;  as  explained  above).  Types  of  products  are  further 
subdivided,  where  appropriate,  into  architectural  subtypes.  These  subtypes  are  explained  in 
the  appendix  of  this  report,  Architectural  Attribute  of  Tools. 


7  The  August  2000  revision  of  the  Compendium  includes  53  product  descriptions;  see  the 
citation  in  the  bibliography  of  this  paper. 
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Table  1.  Summary  of  Types  of  Tools  by  Vendor  Categories 


Product  Type 

Architecture 

P 

S 

o 

IDR  Director 

Director 

1 

IDR  Support  Tool 

Sensor 

2 

Network  Mapper  +  Vulnerability  Scanner  +  Risk  Analyst 

Sensor 

1 

Network  Monitor 

Sensor 

4 

Sensors-Director 

3 

unknown 

1 

Network  Monitor  +  Infraction  Scanner 

Sensors-Director 

1 

Security  Compliance  Scanner 

Sensor 

1 

2 

Director 

1 

Sensors  - 
Director 

1 

Suite  of  Monitors  (web  access,  LAN  packets,  tracing) 

Sensor 

1 

System  Monitor 

Sensor 

2 

2 

Sensors-Director 

4 

1 

1 

System  Monitor  +  Vulnerability  Scanner 

Sensors-Director 

1 

System  Monitor  for  Access  Control 

Sensors-Director 

1 

Vulnerability  Scanner 

Sensor 

5 

1 

Vulnerability  Scanner  with  built-in  Infraction  Scanning 

Sensors-Director 

1 

Vulnerability  Scanner  with  built-in  Network  Mapping 

Sensor 

2 

The  tools  listed  in  this  table  go  well  beyond  intrusion  detection,  with  functionality 
extending  into  areas  of  analysis  that  just  several  years  ago  were  untouched  by  the 
commercial  vendors.  Table  2  shows  the  major  functions  of  the  CSMn  products  reported  in 
Table  1  with  a  count  for  each  function  of  how  many  products  provide  that  function.  In  this 
table,  a  product  may  be  counted  more  than  once.  If,  for  example,  it  provides  both 
vulnerability  scanning  and  network  mapping,  it  counts  in  both  categories. 
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Table  2.  Counts  of  Products  Providing  the  Major  Functions 


Function 

P 

S 

O 

IDR  Management 

1 

IDR  Support 

2 

Network  Mapping 

2 

1 

Risk  Analysis 

1 

Network  Monitoring 

4 

6 

Infraction  Scanning 

2 

Security  Compliance 
Scanning 

2 

2 

System  Monitoring 

7 

2 

3 

Vulnerability 

Scanning 

8 

2 

Table  2  shows  evidence  of  the  broadening  functionality  of  the  tools.  Recall  our  comment 
earlier  that  the  primary  providers  generally  have  the  more  recent  tool  offerings8.  Taking  the 
Primary,  Secondary,  and  Other  column  headings  in  the  table  as  a  rough  time  line  extending 
backward  in  time,  we  observe  a  marked  increase  in  the  number  and  diversity  of  functions 
provided.  The  vulnerability  scanning  function  stands  out  in  this  regard.  Older  tools  primarily 
provided  network  and  system  monitoring  capability,  the  core  functions  of  intrusion  detection. 
We  see  now  a  burgeoning  supply  of  cybersecurity  monitoring  capabilities. 

Table  3  displays  the  number  of  products  in  the  three  major  architectural  types,  organized 
by  provider.  The  three  architectural  types  are  Sensor/ Agent  (includes  Sensor  and  Agent  types 
in  Table  1),  Director,  and  Sensors/Agents-Director  (includes  Sensors-Director  and  Agents- 
Director  types  in  Table  1). 


One  of  the  Security  Compliance  Scanning  tools  listed  under  the  Other  column  is,  in  fact, 
a  new  tool  recently  made  available  (i.e.,  Microsoft’s  Security  Compliance  Monitor). 
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Table  3.  Numbers  of  Products  by  Architectural  Types 


Architecture 

P 

S 

O 

Sensor 

11 

2 

11 

Director 

1 

1 

Sensors-Director 

11 

2 

1 

The  data  of  Table  3  indicate  an  increasing  use  of  the  Sensors-Director  architecture  in 
more  recent  products.  This  certainly  agrees  with  one’s  intuitive  notion  of  how  such  products 
might  evolve.  With  the  use  of  the  Sensors-Director  architecture  comes  improved  scalability 
of  the  products  in  some  cases.  For  example,  when  a  Director  unit  can  accept  reports  and 
inputs  from  other  Director  units,  the  possibility  exists  for  a  hierarchical  arrangement  of 
CSMn  capability  with  wide  coverage  at  the  “leaf’  level. 

Comparison  of  Tools  to  Issues  Identified  in  Earlier  Reports 

We  can  get  a  good  sense  of  how  the  state  of  the  art  is  evolving  by  examining  issues 
identified  earlier  in  light  of  today’s  capabilities.  In  Table  4  we  list  each  of  the  issues 
identified  in  the  three  reports  summarized  earlier  in  this  paper.  The  source  of  each  issue  is 
indicated  by  a  short  reference  as  follows: 

•  NISTB:  the  National  Info-Sec  Technical  Baseline  report 

•  H&A:  the  report  of  Hill  and  Aguirre 

•  IDSR:  the  Intrusion  Detection  Subgroup’s  Report 

Beside  each  issue,  we  provide  commentary  based  on  our  understanding  of  available 
COTS  tools. 


Table  4.  Comparison  of  Tools  to  Issues  Identified  in  Earlier  Reports 


Issue 

Commentary 

(NISTB)  Detection  of  known  intrusion 
techniques 

The  NISTB  report  indicated  that  the  tools 
could  reliably  detect  a  substantial  number  of 
known  intrusion  techniques.  Today  that 
number  is  in  the  range  of  four  to  five 
hundred  known  patterns  for  pattern¬ 
matching  network  monitors. 
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Issue 

Commentary 

(NISTB)  False  positives 

(IDSR)  False  alarms  are  a  major 
performance  problem  in  current  IDSs 

We  have  seen  no  evidence  of  significant 
improvement  in  avoiding  false  positives  at 
the  detection  level;  this  problem  may  be 
ameliorated  by  improved  capability  for  user 
customization  and  analysis.  The 
performance  may  now  be  more  of  an 
analysis  problem  than  a  capture  problem. 

(NISTB)  False  negatives 

There  are  known  cases  of  false  negatives 
occurring  with  esoteric  attacks.  The 
situation  has  improved,  however,  for  run-of- 
the-mill  attacks  as  the  tools  increasingly 
provide  comprehensive  coverage  of  known 
techniques  and  increasingly  more  frequent 
and  user-friendly  pattern  updates. 

(NISTB)  Limiting  the  effect  of  automated 
responses  to  prevent  tool- induced  failures; 
providing  safeguards  in  the  automated 
response  system  against  false  positives  and 
enemy-induced  erroneous  responses;  and 
using  the  response  system  to  push  the  point 
of  attack  deflection  back  toward  the  attack 

source. 

The  NISTB  report  indicated  that  these  areas 
were  essentially  unaddressed  at  that  time 
(December  1997).  Current  tools  show  no 
direct  evidence  of  progress  in  these  areas. 
However,  they  generally  provide  user 
interfaces  that  allow  some  customization, 
which  may  make  it  easier  for  an 
administrator  or  operator  to  have  better 
control  over  these  factors. 

(NISTB)  Testing  is  lacking 

(IDSR)  Standards,  testing,  and  validation 
procedures  are  needed  to  enable 
verification  of  IDSs’  capabilities 

There  are  no  testing  tools  in  the  arsenal  of 
COTS  CSMn  tools,  neither  built-in  nor  as 
part  of  a  suite,  with  one  exception  among 
the  products  examined  in  the  Compendium 
[4].  The  vendor  of  one  recent  vulnerability 
scanning  product  provides  software  for 
setting  up  a  fake  DNS  server  to  enable  the 
scanning  product  to  check  for  the  DNS 
server  cache-overflow  vulnerability;  the 
scanner  also  comes  with  a  scripting 
language  that  allows  users  to  create 
specialized  network  packets  for 
vulnerability  testing. 
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Issue 

Commentary 

(NISTB)  Damage  assessment  and  recovery 
mechanisms  are  lacking  in  the  vast  majority 
of  systems 

(IDSR)  No  systems  appeared  to  provide  an 
automated  damage  assessment  and  response 
capability 

There  has  been  little  change  in  this  area;  the 
COTS  tools  we  have  studied  generally 
provide  no  damage  assessment  and  recovery 
capabilities.  Two  products  provide  some 
capability  for  automatic  repair  of  illicit 
changes.  One  uses  a  fairly  general  technique 
that  can  interface  to  a  variety  of  network 
management  systems. 

(NISTB)  Scalability  of  IDR  systems  is 
severely  limited 

(IDSR)  Many  of  the  deployed  IDSs  are 
unable  to  scale  to  the  large  environments 
characterized  by  thousands  of  network 
nodes,  which  limits  their  ability  to  detect 
intrusions  effectively  across  different 
platforms,  applications,  networks,  and 
infrastructures. 

Scalability  has  improved,  especially 
because  of  the  increasing  use  of  the 
Sensors-Director  architecture.  Three 
vendors  explicitly  address  this  area, 
providing  security  compliance  scanning, 
system  monitoring,  and  network  monitoring 
capabilities  that  scale  up  to  enterprise-wide 
networks  (e.g.,  WANs). 

The  scale  of  operation  among  the  tools 
available  today  is  improved  over  what  it  is 
was  two  or  three  years  ago;  some  of  today’s 
intrusion  detection  tools  can  cover  large 
environments  of  thousands  of  nodes.  As 
these  tools  come  into  use,  the  situation 
noted  in  IDSR  should  gradually  change. 

(NISTB)  Traceback  capability 

None  of  the  intrusion  detection  tools 
examined  provided  any  capability  in  this 
area. 

(H&A)  No  common  terminology  or 
taxonomy  exists  for  discussing  intrusion 
detection  techniques  or  implementation 

The  situation  has  improved;  the  vendors  of 
the  COTS  tools  examined  in  this  study  are 
essentially  speaking  the  same  language  in 
describing  their  offerings. 

12 


Issue 

Commentary 

(H&A)  The  collection  function  in  intrusion 
detection  systems  needs  improvement:  it 
should  address  new  physical  media  such  as 
ATM  and  Fast  Ethernet  and  provide  more 
flexibility  in  signature  specification. 

(IDSR)  Current  IDSs  can  detect  only  what 
they  have  been  preprogrammed  to  detect, 
like  virus  detectors;  this  limits  their 
effectiveness  in  the  face  of  new 
attacks. 

The  intrusion  detection  tools  examined  in 
this  study  generally  cover  Ethernet,  Fast 
Ethernet,  and  FDDI  network  topologies; 
none  directly  address  ATM.  Some  tools 
allow  the  user  to  add  signature 
specifications  or  rules. 

In  addition,  many  vendors  are  now 
providing  updates  on  a  regular  basis  (at 
least  monthly);  the  trend  is  toward 
automatic  updates  via  the  Internet  as  new 
attacks  are  codified;  typically  today  there  is 
an  e-mail  notification  and  updates  can  be 
downloaded  manually. 

(H&A)  Better  analysis  support  is  needed: 
improved  expert  systems  should  reduce  the 
false  alarm  rate;  and  tools  that  can  correlate 
activity  from  distributed  sources  and 
activity  directed  toward  distributed  targets 
are  crucial  to  address  issues  of  information 
warfare. 

Some  progress  has  been  made:  a  number  of 
tools  are  able  to  collect  data  from 
distributed  sources  for  presentation  to  the 
user;  some  correlation  capability,  to 
correlate  multiple  attacks  and  to  correlate 
attacks  with  vulnerabilities,  is  now 
available. 

(H&A)  There  is  growing  recognition  that 
there  would  be  high  utility  in  integrating  the 
output  of  different  entities  involved  in 
network  security,  including  routers, 
firewalls,  proxies,  and  host-based  and 
network-based  IDSs. 

A  discernible  trend  in  this  direction  has 
developed  among  the  COTS  products. 

Those  products  that  are  bundled  in  suites 
typically  can  integrate  the  output  of 
multiple,  different  sensors  at  a  manager 
station. 

(IDSR)  Detection  and  response  need  to 
occur  in  real  time  to  limit  potential  damage; 
current  IDSs  fall  far  short  of  this  ideal 

Although  responses  in  real  time  are  limited 
in  nature,  they  can  occur  rapidly  enough  to 
avoid  or  limit  damage  (e.g.  shutting  down  a 
session,  terminating  a  process,  shunning  a 
connection). 

Government  Products 

We  base  our  findings  in  this  section  on  product  information  gathered  primarily  from 
providers’  web  sites,  as  reported  in  the  Compendium  [4].  The  Compendium  includes 
information  about  three  products  available  from 
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•  Air  Force  Information  Warfare  Center  (AFIWC/AFCERT),  providing  Automated 
Security  Incident  Measurement  (ASIM),  Version  2.0 

•  Lawrence  Livermore  National  Laboratory,  providing  Network  Intrusion  Detector 
(NID),  Version  2.1 

•  DISA  Information  Assurance  Support  Environment  (IASE),  providing  Joint  Intrusion 
Detection  System  (JIDS),  Version  2.0.3 

JIDS  is  the  DoD  version  of  NID,  providing  essentially  the  same  functionality  as  NID. 
These  tools  are  considered  network  monitors  by  their  providers.  In  the  terminology  of  this 
report,  ASIM  running  in  batch  mode  is  considered  an  infraction  scanner,  in  real-time  mode  a 
network  monitor.  NID  (and  JIDS)  are  architecturally  sensors;  ASIM  uses  an  agents-director 
architecture. 

These  tools  are  very  similar  in  functionality,  essentially  performing  an  intrusion  detection 
function  by  analyzing  Ethernet  and  LDDI  packets.  The  major  difference  is  that  NID  (and 
JIDS)  sensors  deal  with  a  single  security  domain  while  the  ASIM  director  unit  deals  with 
multiple  security  domains,  each  of  which  has  an  ASIM  sensor  that  reports  to  the  single 
director  unit. 

What  do  we  learn  about  the  state  of  the  art  from  these  tools?  The  fact  that  they  came  into 
being  in  the  first  place  indicates  that  a  need  existed  that  could  not  be  met  by  commercial 
products  within  the  budgetary  constraints  in  place  at  that  time.  This  appears  to  continue  to  be 
a  factor,  especially  with  regard  to  the  scale  on  which  ASIM  operates.  Aside  from  this 
consideration,  the  GOTS  products  appear  to  offer  no  capability  that  is  technically  superior  to 
that  of  the  COTS  products  and,  in  fact,  appear  now  to  be  lagging  in  providing  some 
significant  features,  such  as  SNMP  traps  for  alerting,  automatic  updating  of  signatures,  and 
user-customization  capabilities. 

Research  Efforts 

In  this  section  we  use  information  gathered  from  researchers’  web  sites,  as  reported  in  the 
Compendium  [4].  The  research  efforts  reported  in  the  Compendium  are 

•  Automated  Intrusion  Detection  Environment  (AIDE)  Advanced  Concept  Technology 

Demonstration  (ACTD) 

•  Autonomous  Agents  for  Intrusion  Detection  (AAFID) 

•  Common  Intrusion  Detection  Framework  (CIDF) 

•  DARPA  Intrusion  Detection  Evaluation 

•  Distributed  Intrusion  Detection  System  (DIDS) 

•  Event  Monitoring  Enabling  Responses  to  Anomalous  Live  Disturbances 

(EMERALD) 


14 


•  Extensible  Prototype  for  Information  Command  and  Control  (EPIC^) 

•  Graph-based  Intrusion  Detection  System  (GrIDS) 

•  Next-Generation  Intrusion  Detection  Expert  System  (NIDES) 

•  Spitfire 

In  Table  5,  we  briefly  state  the  main  thrust  of  each  effort  and  comment  on  it  with  respect 
to  how  it  participates  in  defining  the  state  of  the  art. 


Table  5.  Main  Thrusts  of  Research  Efforts 


Project 

Thrust 

Comment 

Automated 

Intrusion 

Detection 

Environment 

(AIDE) 

Advanced 

Concept 

Technology 

Demonstration 

(ACTD) 

This  5-year  technology  demonstration 
program  focuses  on  integrating  data 
from  network  management  and 
information  protection  systems  in  order 
to  provide  automated,  integrated, 
tactical  warning  and  attack  assessment. 
The  program  has  set  three  objectives: 

•  Develop  an  architecture  for 
integration,  analysis,  and  warning  of 
IW  attacks 

•  Incorporate  current  and  maturing 
intrusion  sensing  tools  using  expert 
systems  technology  for 
management  of  distributed  systems 

•  Correlate  intrusion  events  at  local, 
regional,  and  global  command 
levels  to  improve  the  probability  of 
detection  and  identification  of  IW 

events 

The  current  implementation 
of  the  objective  tool  is 
EPIC^.  See  the  description 
of  EPIC2  for  information 
about  the  current  properties 
of  the  objective  tool. 

Autonomous 
Agents  for 
Intrusion 
Detection 
(AAHD) 

This  project  is  experimenting  with  a 
distributed  architecture,  within  which 
various  types  of  autonomous  agents  can 
be  accommodated. 

Contributing  to  defining  the 
state  of  the  art  in  distributed 
CSMn  systems. 
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Project 

Thrust 

Comment 

Common 

Intrusion 

Detection 

Framework 

(CIDF) 

This  is  a  standards  effort,  by 
consortium,  to  develop  protocols  and 
application  programming  interfaces  so 
that  intrusion  detection  products  can 
interoperate  and  components  of  them 
can  be  reused  in  other  systems. 

Defining  the  state  of  the  art 
in  this  area. 

DARPA 

Intrusion 

Detection 

Evaluation 

This  effort  is  to  develop  testing  and 
evaluation  standards;  it  is  collecting  and 
distributing  the  first  standard  corpus  for 
evaluation  of  intrusion  detection 
systems,  both  host-based  and  network- 
based. 

Defining  the  state  of  the  art 
in  this  area. 

Distributed 

Intrusion 

Detection 

System  (DIDS) 

This  effort  was  to  develop  intrusion 
detection  for  large,  distributed  networks 
employing  multiple  agents,  both  host- 
based  and  network-based,  reporting  to  a 
centralized  director,  which  would  then 
be  able  to  correlate  inputs  so  that  a 
single  user  could  be  tracked  across 
multiple  networks. 

The  information  available 
on  this  project  is  from 

1991.  At  that  time,  this 
effort  was  defining  the  state 
of  the  art.  It  has  apparently 
since  been  overtaken  by 
events. 

Event 

Monitoring 
Enabling 
Responses  to 
Anomalous 

Live 

Disturbances 

(EMERALD) 

This  project  is  attempting  to  develop  a 
distributed  scalable  tool  suite  for 
tracking  malicious  activity  through  and 
across  large  networks.  Using  distributed 
monitors,  it  would  provide  a  global 
detection  and  response  capability  to 
counter  attacks  occurring  across  an 
entire  network  enterprise. 

The  scale  being  attempted 
by  this  effort  should  raise 
issues  that  will  help  to 
refine  the  state  of  the  art  in 
systems  that  can  span 
global  networks.  This  effort 
is  similar  to  others  in 
progress  and  is  a  follow-on 
to  NIDES. 
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Project 

Thrust 

Comment 

Extensible 
Prototype  for 
Information 
Command  and 
Control 
(EPIC2) 

This  project’s  goals  are  to  achieve 
interoperability,  integration,  and 
coordination  of  intrusion  control  tools 
in  a  manner  that  can  scale  to  global 
networks.  The  main  thrust  is  to  employ 
an  expert  system  that  can  interact  with 
diverse  agents.  Scalability  is  achieved 
by  hierarchical  arrangement  of  the 
expert  systems. 

Defining  the  state  of  the  art 
in  the  use  of  expert 
systems. 

Graph-based 

Intrusion 

Detection 

System 

(GrIDS) 

This  project  is  oriented  toward 
detecting  large-scale  automated  attacks 
on  networked  systems.  The  main  idea  is 
to  build  activity  graphs  in  which  nodes 
represent  hosts  and  edges  represent 
network  activity  among  them.  The 
detection  technique  is  to  compare  a 
graph  to  a  known  pattern  of  intrusive 
activity. 

Attempting  to  create  a  new 
detection  method. 

Next- 

Generation 

Intrusion 

Detection 

Expert  System 
(NIDES) 

This  project  created  a  system  monitor, 
using  a  Sensors-Director  architecture, 
employing  both  pattern  matching  and 
statistical  deviation  detection  methods. 

This  system  has  been 
overtaken  by  events;  see 
successor  EMERALD. 

Spitfire 

This  effort  produced  an  intrusion  alert 
manager  implemented  in  a  client/server 
architecture.  The  client:  is  a  GUI 
providing  access  to  data  stored  on  the 
server.  The  server:  provides  access  to 
an  Oracle  database  that  stores  intrusion 
alerts,  which  can  come  from  diverse 
sensors,  and  vulnerability  and  tool 
information. 

This  effort  helped  to  push 
the  state  of  the  art  in 
providing  a  user-friendly 
interface  for  managing 
intrusion  alerts  and  in 
integrating  vulnerability 
and  tool  (e.g.,  vulnerability 
scanners)  information  into 
the  alert-operators 
environment. 
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Summary 

We  capsulize  here  the  state  of  the  art  as  we  have  reviewed  it  in  this  paper. 

Table  6.  Condensation  of  State  of  the  Art  in  CyberSecurity  Monitoring 


Topic 

State 

Detection  reliability 

Intrusion  detectors  can  detect  significant  number  of  intrusions 
and  short-term  changes  in  behavior;  but  false-positives  and  false- 
negatives  can  still  be  a  problem.  The  situation  has  improved, 
however,  for  run-of-the-mill  attacks  as  the  tools  increasingly 
provide  comprehensive  coverage  of  known  techniques  and 
increasingly  more  frequent  and  user-friendly  pattern  updates. 
Vulnerability  scanners  can  detect  significant  numbers  of 
vulnerabilities;  automatic  updating  of  vulnerability  libraries  helps 
to  alleviate  the  problem  of  detecting  new  vulnerabilities. 

Detection  of  new 
attacks 

Most  IDSs  detect  only  what  they  have  been  preprogrammed  to 
detect;  some  progress  in  this  area  is  now  being  made  in  COTS 
tools,  some  of  which  can  detect  variations  on  preprogrammed 
patterns;  automatic  updates  to  pattern  libraries,  which  some 
vendors  have  begun  to  provide,  also  help  in  this  area. 
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Topic 

State 

Reaction  capability, 
including  damage 
assessment  and 
recovery,  and  real¬ 
time  reaction 

Current  products  are  limited  to  providing  input  to  the  decision 
maker;  there  is  no  general  capability  for  automated  damage 
assessment  and  repair,  but  two  commercial  products  do  have 
some  ability  to  fix  illicit  changes  automatically  (one  can  repair 
configuration  changes,  another  uses  a  general  response  module 
that  can  interface  to  other  management  tools);  useful  automated 
response  capability  beyond  notification  depends  on  addressing 
several  issues  such  as  how  to  ensure  that  the  response  system 
does  not  itself  induce  errors  and  how  to  safeguard  against  false 
positives  and  enemy-induced  erroneous  responses.  Commercial 
tools,  however,  generally  provide  user  interfaces  that  allow  some 
customization,  which  may  make  it  easier  for  an  administrator  or 
operator  to  better  control  these  factors. 

Real-time  reaction  capability  in  monitoring  tools  is  generally 
provided  in  the  form  of  online  alerting,  e-mail  notification, 
connection  termination,  and  SNMP  traps;  a  few  tools  can  interact 
with  firewalls;  using  SNMP  traps,  it  should  be  possible  to 
implement  enterprise-specific  reaction  pohcies. 

Although  responses  in  real  time  are  limited  in  nature,  they  can 
occur  rapidly  enough  to  avoid  or  limit  damage. 

Analysis  (for 
example, 
consolidation  and 
correlation  of 
collected  data) 

COTS  products  have  traditionally  had  little  or  no  capability.  This 
area  has  now  been  addressed  by  at  least  two  vendors.  For 
example,  one  vendor  claims  its  product  relates  vulnerability  data 
about  hosts  with  attack  data  to  show  which  hosts  are  both 
vulnerable  and  being  attacked.  In  addition,  a  number  of  tools  are 
able  to  collect  data  from  distributed  sources  for  presentation  to 
the  user.  Several  of  the  research  projects  we  examined  must 
employ  some  form  of  correlation  technique  to  achieve  their 
goals;  the  AIDE  ACTD  and  EMERALD  projects  directlv  address 
attack-data  correlation. 
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Topic 

State 

Traceback 

capability 

This  is  an  unsolved  problem.  Current  tools  may  have  traceback 
features,  but  they  can  be  spoofed;  traceback  can  require  the 
cooperation  of  independent  network- service  providers,  a 
potential  sticking  point. 

Standards  and 
terminology 

Standards  are  lacking,  but  the  CIDF  and  IETF9  initiatives  are 
promising.  Terminology  appears  to  be  stabilizing  with  generally 
common  understanding  of  terms. 

Network  topologies 

One  can  generally  expect  that  available  tools  can  operate  in 
Ethernet,  Fast  Ethernet,  and  FDDI  environments.  No  tools  deal 
directly  with  ATM. 

Testing  and 
validation 

Testing  and  validation  are  inadequate  at  present  but  there  is  at 
least  one  initiative  (DARPA  Intrusion  Detection  Evaluation)  to 
develop  benchmarks  for  testing  intrusion  detection  products  and 
at  least  one  vendor  is  providing  specialized  testing  capability 
with  its  vulnerability  scanning  product. 

Scalability 

Scalability  is  lacking  in  some  products  for  which  scalability 
could  be  a  feature  (for  many  products,  scalability  is  simply  not  an 
issue);  some  vendors  (at  least  three)  are  beginning  to  address  this 
area  using  distributed,  hierarchical  architectures;  note  also  that 
the  scale  of  applicability  of  tools  in  general  can  be  quite  large; 
some  emphasis  on  scalability  is  apparent  in  the  research  projects 
we  examined  (EMERALD,  EPIC2,  and  GrIDS). 

Integration  of  tools 
into  comprehensive 
CSMn  systems 

Rudimentary  capabilities  are  beginning  to  show  up  in  COTS 
products,  typically  as  a  bundled  suite  of  previously  independent 
tools  by  the  same  vendor;  this  is  an  area  that  GOTS  products 
developed  earlier  than  the  commercial  vendors  and  current 
government-funded  efforts  continue  to  lead  in  this  area. 

Performance 

False  alarm  rates  can  be  high,  especially  if  an  intrusion  tool  is  not 
tailored  to  the  environment  in  which  it  operates,  creating  a 
performance  problem  for  the  analyst;  many  packet  monitoring 
systems  are  not  designed  for  networks  operating  at  100  megabits 
per  second  and  higher. 

9  Effective  November  23,  1998,  a  new  working  group,  called  the  Intrusion  Detection 
Working  Group,  was  formed  in  the  Security  Area  of  the  IETF.  The  purpose  of  the 
Intrusion  Detection  Working  Group  is  to  define  data  formats  and  exchange  procedures 
for  sharing  information  of  interest  to  intrusion  detection  and  response  systems,  and  to 
management  systems  which  may  need  to  interact  with  them. 


20 


Topic 

State 

Research  and 
development  in  U.S. 
Government  funded 
efforts 

Correlation  of  data  from  several  sources  (e.g.,  EMERALD  and 
GrIDS)  and  scalabilitv  (EMERALD,  EPIC2,  and  GrIDS)  are 
being  worked  on.  There  appears  to  be  no  research  on  damage 
assessment  and  recovery. 

GOTS  systems 

These  systems  continue  to  be  an  important  part  of  the  protection 
arsenal  for  the  military,  especially  for  integrating  tools  and  to 
achieve  adequate  scope,  that  is,  the  ability  to  deal  with  a  large 
protection  domain.  This  was  apparently  initially  the  case  because 
commercial  tools  could  not  provide  the  capability  needed;  now,  it 
appears  to  be  more  a  matter  of  economics  than  technology. 

Updates 

Most  vendors  now  provide  updates  for  their  intrusion  detection 
and  vulnerability  scanning  tools.  In  many  cases,  updates  are 
product  updates  available  to  existing  customers.  In  some  cases, 
updates  are  issued  on  a  regular  basis,  typically  available  via 
download  or  supplied  on  a  floppy.  In  one  case,  automatic, 
periodic  updates  are  available  via  the  Internet. 
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Appendix 

About  CyberSecurity  Monitoring  Systems 

The  technology  of  CyberSecurity  Monitoring  is  based  on  observation,  experience,  and 
classification  of  attacks,  vulnerabilities,  and  countermeasures.  Two  key  aspects  of  the 
technology  today  are  the  methods  of  detection  and  the  types  of  tools  available. 

Methods  of  Detection 

There  are  many  terms  used  for  describing  methods  of  detection,  but  all  the  methods  we 
have  seen  described  fall  into  just  one  of  two  categories10 

•  Statistical  deviation  detection 

•  Pattern  matching  detection 

Statistical  Deviation  Detection 

In  this  approach,  the  CSMn  tool  looks  for  deviations  from  statistical  measures.  A 
baseline  of  values  is  defined  for  subjects  and  objects  such  as  users,  groups,  workstations, 
servers,  files,  and  network  adapters.  One  can  use  historical  data,  simple  counting,  or  expected 
values  to  establish  the  baseline.  As  activities  being  monitored  occur,  the  CSMn  tool  updates 
a  list  of  statistical  variables  for  each  subject  or  object  of  interest.  For  example,  the  engine 
might  count  the  number  of  files  read  by  a  particular  user  over  a  given  period.  This  method 
treats  any  unacceptable  deviation  from  expected  values  as  an  anomaly.  For  example,  when 
the  number  of  files  read  by  a  particular  user  over  a  given  period  exceeds  the  expected  value 
for  that  period,  the  CSMn  tool  declares  a  potential  anomaly. 

Practitioners  use  various  terms  for  and  explanations  of  this  type  of  detection.  Some  are 

•  Anomaly  detection:  detecting  deviation  from  a  normal  pattern  of  usage,  as  with  an 
insider’s  use  of  an  enterprise  network 


10In  his  recent  book,  Escamilla  also  identifies  two  types  of  detection;  he  calls  them  statistical 
anomaly  detection  and  pattern  matching  detection,  referring  to  them  as  “IDS  engine 
categories.”  We  use  the  terms  to  encompass  vulnerability  scanners  and  policy  compliance 
monitors  as  well.  In  his  recent  book,  Amoroso  discusses  five  methods  used  in  practical 
intrusion  detection;  the  five  methods  are  not  mutually  exclusive  and  are  often  used  in 
combination;  this  is  a  different  and  very  useful  way  to  view  detection  methods  and 
Amoroso’s  discussion  sheds  light  on  how  people  and  their  automated  tools  actually 
operate  [7]. 
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•  Statistical  anomaly:  detection  is  based  on  the  assumption  that  users  and  networks 
exhibit  predictable  patterns  of  behavior  from  which  they  do  not  deviate  significantly 
over  short  periods  of  time;  a  deviation  from  normal  indicates  a  possible  attack. 

•  Rule-based  detection:  detection  based  on  a  library  of  statistical  descriptions  of 
acceptable  behaviors. 

Pattern  Matching 

In  this  approach,  the  CSMn  tool  compares  activity  to  stored  patterns  that  model  attacks  or 
unacceptable  states.  Known  attacks  or  types  of  attacks  as  well  as  proper  configurations  or 
system  security  policies  are  modeled  as  patterns  of  data.  Patterns  can  be  composed  of  single 
events,  sequences  of  events,  thresholds  of  events,  or  expressions  using  AND,  OR,  and  NOT 
operators11.  This  method  treats  any  activity  or  state  that  matches  a  pattern  as  a  potential 
problem. 

Practitioners  use  various  terms  for  and  explanations  of  this  type  of  detection.  Some  are 

•  Misuse  detection:  detecting  attempted  exploitation  of  a  specific  vulnerability 

•  Signature  detection:  detecting  specific  characteristics  of  a  transmission  or  of  the 
message  being  received 

•  Rule-based  detection:  detection  based  on  a  library  of  known  attack  patterns, 
unauthorized  activity,  or  unacceptable  system  parameter  values. 

In  addition,  recent  work  done  at  Lincoln  Lab  might  be  called  “privilege  anomaly 
detection”  12.  This  kind  of  detection  is  based  on  analysis  of  audit  logs.  Analysis  deduces  an 
intrusion  as  follows:  if  a  process  has  privilege  “x”  but  there  is  nothing  in  the  audit  log 
showing  how  it  got  the  privilege  in  an  authorized  manner,  the  analyzer  deduces  that  the 
privilege  was  attained  in  an  unauthorized  manner. 

Use  of  the  Methods 

The  two  methods  of  intrusion  or  abnormality  detection  can  be  applied  to  both  network- 
based  detectors  and  host-based  detectors.  Traditionally,  however,  network-based  monitors 
used  pattern  matching  and  host-based  monitors  used  statistical  deviation  detection.  Network- 
based  detectors  normally  are  sited  at  a  place  in  the  network  at  which  all  the  traffic  going  into 


11  As  Escamilla  notes,  negation  could  be  used  for  detecting  unacceptable  events  but  it  might 
introduce  computational  complexity  since  it  could  require  looking  for  “everything  but 
this  event”  [5].  More  likely,  negation  will  be  useful  for  modeling  data  values  that  are  out 
of  bounds. 

12  We  have  invented  this  term  for  the  type  of  detection  reported  here. 
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and  coming  out  of  a  domain  of  protection  can  be  monitored.  Network  traffic  being  the  only 
input  to  a  network-based  detector,  the  detector  does  not  depend  on  data  from  or  cooperation 
with  other  systems  on  the  network.  It  examines  packets  going  past  it  looking  for  patterns  in 
selected  data  that  may  indicate  a  problem.  A  host-based  detector,  normally  sited  on  the  host 
it  is  to  protect,  can  look  for  attacks  related  to  specific  vulnerabilities  of  its  host,  can  analyze 
local  system  logs,  and  can  monitor  local  user  activity.  Host-based  tools  may  use  the  Sensors- 
Director  architecture,  in  which  case  the  Sensor  performs  the  collection  function  (e.g.,  gather 
data  from  event  logs),  may  do  some  preliminary  analysis  or  filtering  of  the  data,  and 
transmits  the  selected  data  to  the  Director,  where  analysis,  correlation,  storage,  and  other 
functions  may  be  performed. 

Types  of  Tools 

Two  traditional  terms  for  intrusion  detection  systems  are  “host-based”  and  “network- 
based.”  These  terms  are  likely  to  disappear  from  usage  as  the  variety  of  CyberSecurity 
Monitoring  systems  available  today  no  longer  fit  this  simplistic  taxonomy.  More  current 
terminology  has  been  used  by  Escamilla,  who  identifies  three  main  categories  of  intrusion 
detection  tools  [5]. 

•  Scanners:  A  scanner  is  an  IDS  that  periodically  looks  for  vulnerabilities  that  might 
open  a  system  to  exploitation  by  an  intruder  or  for  evidence  of  intrusions  after  they 
have  occurred.  Scanners  can  run  directly  on  the  target  or  can  scan  targets  from  a 
remote  location. 

•  System-level  monitors:  System-level  monitors  look  for  evidence  of  misuse  and 
intrusion  in  real  time.  They  may  use  pattern  matching  or  statistical  deviation 
detection.  System-level  monitors  gather  data  from  the  target  systems  and  typically 
send  the  data  to  a  central  director  component  that  analyzes  the  data. 

•  Network  sniffers:  A  network  sniffer  examines  all  incoming  network  packets  on  its 
subnet  looking  for  indications  of  intrusion  attempts.  Typically,  a  network  sniffer  runs 
on  a  computer  on  the  subnet  whose  network  adapter  operates  in  promiscuous  mode13. 
A  network  sniffer  can  also  run  on  some  remote  computer  and  communicate  with  a 
router  that  provides  it  network  traffic  information  for  the  subnet  being  sniffed14. 

We  can  distinguish  between  scanners  that  look  for  vulnerabilities  and  scanners  that  look 
for  infractions.  Some  practitioners  mean  both  vulnerabilities  and  infractions  when  they  use 


13  In  promiscuous  mode,  a  network  adapter  sees  all  packets  on  its  subnet.  In  normal  usage, 
such  an  adapter  captures  and  passes  up  to  its  host  those  packets  intended  for  its  host. 

14  NetRanger,  by  Cisco,  provides  a  configuration  such  as  this,  in  addition  to  being  able  to 
run  in  standalone  mode  with  a  promiscuous  mode  adapter. 
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the  term  “vulnerabilities.”  That  is,  an  infraction,  such  as  a  violation  of  access  policy  by  an 
intruder  who  gains  root  access  to  a  UNIX  system,  is  looked  upon  as  a  vulnerability.  Here  the 
vulnerability  is  represented  by  the  changes  in  the  system  caused  by  the  infraction.  The 
assumption  is  being  made  that  the  infraction  actually  did  cause  one  or  more  changes  in  the 
system.  However,  the  assumption  may  not  always  hold.  The  infraction  of  the  example  just 
given  may  compromise  data  through  a  read  operation,  leaving  everything  as  it  was  before  the 
infraction,  except  for  entries  that  may  have  been  made  in  logs  and  updates  to  system  data 
such  as  “last  time  accessed”  data  for  the  file  that  was  read.  The  change  in  “last  time 
accessed”  data  most  likely  does  not  represent  a  vulnerability  in  the  system.  We  will 
distinguish  between  scanning  for  vulnerabilities  and  scanning  for  infractions.  An  infraction 
scan  could  discover  that  an  intrusion  occurred  by  examining  audit  logs,  even  though  there 
may  be  no  other  direct  evidence  of  the  intrusion. 

A  difference  between  an  infraction  scanner  and  a  system-level  monitor,  as  pointed  out  by 
Escamilla,  is  that  the  scanner  operates  periodically  while  the  monitor  operates  in  real-time. 
Also,  one  may  detect  intrusions  not  noticed  by  the  other. 

Although  in  theory  a  network  sniffer  could  operate  in  nonreal  time,  in  practice,  sniffers 
operate  in  real  time.  We  will  treat  all  sniffers  as  operating  in  real  time  unless  we  come  across 
a  product  that  does  not. 

We  use  an  extended  terminology  to  characterize  the  type  of  a  CSMn  product  based  on  the 
foregoing  and  the  following  considerations: 

•  Some  tools  are  integrated  integrated  intrusion  detection  and  reaction  systems.  EPIC2 
and  ISS’s  SAFEsuite  Decisions  are  examples.  We  call  such  tools  Intrusion  Detection 
and  Reaction  (IDR)  Directors. 

•  We  wish  to  include  security  compliance  tools. 

•  We  wish  to  distinguish  analysis  engines  as  an  important  class  of  tools. 

•  We  wish  to  include  support  tools  that  provide  data  for  anomaly  analysis. 

Thus,  we  recognize  the  following  types  of  tools  (listed  alphabetically): 

•  Analysis  Engine:  An  analysis  engine  receives  inputs  from  a  variety  of  sources  (e.g., 
intrusion  detectors,  vulnerability  scanners,  etc.),  possibly  from  widely  distributed 
sources,  and  performs  an  analysis  on  the  aggregated  data  to  discover  one  or  more 
things  such  as  widely  distributed  attacks,  distributed  but  coordinated  attacks,  patterns 
of  vulnerabilities,  and  so  forth. 
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•  Intrusion  detection  and  reaction  director  (IDR  Director  or  IDRD):  An  IDRD 
integrates  the  functionality  of  two  or  more  IDR  tools;  these  tools  may  be  of  the  same 
type  or  of  different  types.  For  example,  an  IDRD  may  integrate  the  functionality  of 
many,  identical  network  monitors  or  it  may  integrate  the  functionality  of  a  system 
monitor  and  a  vulnerability  scanner.  The  IDRD  provides  an  interface  for  managing 
IDR  tools  and  their  interactions.  Products  in  this  category  may  range  widely  in  degree 
of  integration.  At  a  minimum,  a  system  in  this  category  provides  a  single  interface  to 
two  or  more  instances  of  the  same  type  of  tool  or  to  two  or  more  types  of  tools  that 
are  interrelated  at  least  via  the  view  presented  to  the  user.  Very  capable  IDRDs 
include  intrasystem  communications  among  multiple  instances  and  types  of  tools. 

•  Intrusion  detection  support  tool:  This  kind  of  tool  does  not  itself  perform  intrusion 
detection  functions  but  gathers  information  that  could  be  used  to  detect  intrusions. 
Tools  of  this  type  might  collect  audit  data  from  hosts  or  data  from  network  packets, 
store  the  data  in  a  database,  and  make  it  available  in  some  user-friendly  form. 

•  Infraction  scanner:  An  infraction  scanner  periodically  looks  for  evidence  of 
infractions,  including  intrusions  by  outsiders  and  violations  of  policy  by  insiders. 

•  Network  monitor:  A  network  monitor  looks  for  evidence  of  attempted  misuse  or 
intrusion  in  real  time  by  examining  data  from  network  packets. 

•  Security  compliance  scanner:  A  security  compliance  scanner  periodically  examines 
the  settings  of  system  parameters  that  are  relevant  to  the  security  of  the  system  to 
ensure  that  they  comply  with  a  preset  policy. 

•  System  monitor:  A  system  monitor  looks  for  evidence  of  misuse  and  intrusion  in  real 
time  by  examining  data  from  the  target  system  and/or  data  in  network  packets 
entering  the  system. 

•  Vulnerability  scanner:  A  vulnerability  scanner  periodically  looks  for  vulnerabilities 
that  might  make  a  system  susceptible  to  exploitation. 

Architectural  Attribute  of  Tools 

One  of  the  attributes  used  in  the  Compendium  for  describing  tools  is  called  Architecture 
[4].  The  Compendium  explains  the  possible  values  of  this  attribute  as  follows15: 

•  Sensor:  A  Sensor  is  a  software/hardware  component  that  one  adds  to  a  system  such  as 
a  server,  workstation,  or  router  to  provide  cybersecurity  management  functions 
specific  to  that  system  or  the  domain  in  which  the  system  is  located.  A  Sensor  may 


15  These  architectural  types,  from  the  revised  Compendium  listed  in  the  bibliography, 
reflect  revised  terminology  which  has  been  applied  throughout  this  paper.  The 
terminological  simplification  is  to  refer  to  both  sensors  and  agents  as  Sensors. 
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operate  independently  of  other  CSMn  capabilities  to  protect  the  system  or  domain.  It 
may  provide  exported  data  or  reports  that  can  be  used  by  other  CSMn  capabilities.  In 
addition,  it  may  operate  under  the  management  of  a  CSMn  Director.  When  a  sensor  is 
specifically  designed  to  operate  with  a  Director,  it  is  often  called  an  Agent. 

•  Director:  A  Director  is  a  software  application  or  a  software  and  hardware  ensemble 
that  performs  storage,  analysis,  reporting,  and/or  command  and  control  functions.  A 
CSMn  Director,  for  example,  may  control  a  hierarchy  of  other  Directors  having 
specific  functions  such  as  intrusion  detection,  vulnerability  scanning,  policy 
checking,  and  so  forth.  An  IDR  Director,  for  another  example,  interacts  with  IDR 
Agents  or  Sensors  within  its  domain.  See  description  of  IDR  Director  under  Types  of 
Tools  above. 

•  Sensors-Director:  self-explanatory 
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